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1 Introduction 


In this paper we study a problem called “On-Off ” in astrophysics and the 
problem of “one signal band and one sideband ” in high energy physics (HEP). 
In the astrophysics version, one points the telescope at ( “On ” ) a potential 
point signal source and observes x counts in a particular amount of observing 
time. Then one points the telescope away from ( “Off ” ) the source to a nearby 
region thought to have no point sources, and observes y counts in an observing 
time that is r times as long as the “On ” observing time. The latter provides 
an estimate of the non-point-source rate 6, which crudely speaking allows for 
a “background subtraction ” in the on-source data. One desires a confidence 
interval for the point source rate /r in the presence of the background rate 
that can be inferred (with some uncertainty) only from y. Complications arise 
because, for weak sources and small counts x and y, the Poisson-distributed 
data is not suitable for trivial formulas coming from Gaussian assumptions. 
So the probability model we are studying is given by 

X ~ Pois{n + b) y ~ Pois{Tb) (1) 


The same probability model arises if the background rate was estimated via 
Monte Carlo, in which case r is related to the number of Monte Carlo runs. 

Statistical Inference for this problem has a long history in HEP and Astron¬ 
omy. It received renewed interest in 1997 by Feldman and Cousins [^, who 
applied their now famous Unified method to the case of a known background 
rate b. A general method for including an uncertainty into the model was 
proposed by Cousins and Highland and was applied to the “On-Off ” 
Conrad et.al. and Tegenfeldt and Conrad 17 


m 


A solution based on inverting the likelihood ratio test was proposed by Rolke 


and Lopez 15 and extended to include uncertainty in the efficiency as well as 
other probability models for Y by Rolke, Lopez and Conrad 16 . This is also 


known as prohle likelihood method, which has a long history in Statistics and 
has been in use for a some time in physics in the MINUIT program when errors 
are calculated with the MINOS method, see James and Roos 
Ma 


12 


et.al. 13 


11 and Li and 


The class TRolke implements this method in ROOT, see Lundberg 


A different approach known as the CLs method was described by Read 14 


and Gan and Kass used the Cousins-Highland prescription to include un¬ 
certainties in the background rate into CLs. This solution is different from the 
others discussed in this paper in that it calculates upper limits only. Finally 
a number of intervals derived via the Bayesian paradigm have been proposed 
as well. 


2 











The problem of significance testing for the signal rate was studied by Cousins, 
Linnemann and Tucker [^, and in principle hypothesis testing and interval 
estimation are the same problem, as one can always invert a hypothesis test 
into an interval procedure and vice versa. In practice this can be difficult, and 
in fact the method favored by Cousins, Linneman and Tucker, namely turning 
the problem into one for a Binomial parameter p, can not be inverted into 
a limit setting method because it is specihc to the test Hq ■. = Q. What is 

missing from the literature is a detailed study of the commonly used methods 
for limit setting in terms of their coverage as well as other properties, and it 
is this study we are undertaking in this paper. 

The software tools commonly used to calculate these limits, for example 
RooStats, are general purpose tools that allow for much more complicated 
probability models, for example multiple channels, uncertainties in the detec¬ 
tion efficiencies etc. Such generality comes at a price as these tools use Monte 
Carlo simulation for the limit calculations. This leads to two problems for our 
study: on the one hand the limits come with errors, often on the order of 
5-10%, and it is not clear how one would determine the correct coverage in 
this case. Moreover, these tools are fairly slow in calculating the limits. This 
is not an issue if just one or even a few limits are needed. We, though, need 
a very large number of them: our study focuses on the small sample case, 
and so we restrict ourselves to /i G [0,20] and b G [0,10]. We also want to 
study a number of different r values, say r G {0.5,1,2}. With these param¬ 
eters possible observations for which we need the limits range from a: = 0 to 
a; = 50 and the same for y. Also we want to study coverage at the nominal 
68%, 90% and 95% levels. This means we need to calculate a total of 23409 
limits for each method. Obviously this requires a very fast way to hnd the 
limits, and in the next section we are developing exact formulas to be able 
to do so. Unfortunately this task becomes hopeless if we tried to incorporate 
further uncertainties, for example in r, into the models. 

It should be noted that all results in this paper are based on exact calculations. 
They therefore do not carry with them any uncertainties due to a hnite number 
of simulation runs. 


2 The Methods 

2.1 Intervals based on Inverting a Likelihood Ratio Test: Rolke-Lopez-Conrad 
(RLC) 


Maybe the most widespread technique for deriving a hypothesis test in Statis¬ 
tics is the likelihood ratio test (LRT). Say we have a probability model /(x; 9) 


3 


and we wish to test Hq : 9 E Qq ys Ha : 9 ^ Qq- Then the LRT is a test based 
on the test statistic 

suPeo /(x;g) 
supe /(x; 9) 

where ©o is some subset of the parameter space 0. 


The reason for the popularity of this approach is two-fold: hrst by the famous 
Neyman-Pearson lemma it is known that in the case of a simple vs. simple 
hypothesis this test is optimal, that is has the highest possible power for a 
given type I error probability a. Even though optimality is not guaranteed 
in more complicated cases, experience has shown that tests (and intervals) 
derived with this method tend to do very well. Also, under some regularity 
conditions in the large sample limit —21ogT(x) has a chi-square distribution. 
This is known as Wilk’s theorem, and a proof can be found in many Statistics 
text books, for example in Casella and Berger [^. 


For our probability model we have 






x\ 


( 3 ) 


so 9 = (/i, b). Here and in what follows r is considered a known constant. We 
want a conhdence interval for p alone, so we test Hq : /i = po vs : /i 7^ po- 
For the denominator of T we need to hnd the maximum likelihood estimators 
(mle), which are p = x — y/r and b = y/r. For the numerator we need to 
maximize /(x, ?/; fi, b) for b alone, treating p as hxed. This leads to 

X X + 2 / - (1 + r)/i + -7(x + ?/ - (1 + r)p)2 + 4(1 + T)y^i 
" =- 


and then the test can be based on 


- 21ogT(x,?/) =2 


log/(x,2/;/i,6) - log/(x,|/;/io,fe) 


( 5 ) 


Replacing a nuisance parameter by the value that maximizes the likelihood 
function while keeping the parameter of interest hxed is known as the prohle 
likelihood method and has a long history in Statistics. 


Unfortunately in our case, at least when /tq = 0, the regularity conditions of 
Wilk’s theorem are not satished. Moreover, we are far from a large-sample 
regime, and so the question arises as to what the null distribution might be. 
This was studied by Rolke and Lopez 15 who showed that the chi-square 


approximation is surprisingly good, and that conhdence intervals derived by 
inverting the likelihood ratio test and using the chi-square approximation have 
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good coverage properties, at least when some ad-hoc adjustments are made in 
the cases where the observed number of events in the signal region is less than 
what is expected from background alone. The RLC method is implemented 
in the Root class Trolke, described in Lundberg et.al. 
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2.2 Feldman-Cousins Unified Method 


In 1997 Feldman and Cousins proposed a limit setting method for the 
model X ~ Pois{fi b) where b is assumed known. The method proceeds as 
follows. Consider the Poisson density 


= ( 6 ) 

x\ 

where x = 0,1,.. and p > 0. Now for all possible observations x and a signal 
rate /i calculate the ratio 

f{FU + b) , . 

f{x]fi + b) 

where p, = max{0, a; — 6} is the maximum likelihood estimator (mle) of /i. 
Rank the possible observations x according to these ratios, resulting in the 
sequence x*fi = 1,2.. Dehne the “acceptance region ” R(/i) by 


{ n n+1 

+ <d,Y.f{x*;fx + b) >d\ (8) 

i=l i=l J 

where cl is the desired conhdence level, for example 0.95 for a 95% conhdence 
interval. In the Statistics literature this methodology for deriving a test (and 
a corresponding limits method) is called the Neyman construction. The con¬ 
hdence interval for fi is comprised of all values of fi such that A{fi) includes 
the observed x. Because in the problem studied here this method leads to a 
simple interval this last step can be done by hnding the endpoints. 

It should be noted that in this case of a Poisson rate with a known back¬ 
ground the limits found by the Feldman-Cousins method can also be derived 
by inverting the corresponding likelihood ratio test. 

How can we extend this method to our more general problem? We will consider 
four options: 
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2.2.1 Feldman-Cousins Confidence Regions (FCR) 

The most obvious solution is to consider confidence regions in (/i, h) space. So 
now we have pairs of points (x, y) and again we find their Poisson probabilities 




x\ y\ 


(9) 


We find the maximum likelihood estimators y = max{0, x —y/r} and b = y/r, 
and calculate the ratios 


fix,y,y,b) 

f{x,y,fl,b) 

Then we rank the points according to this ratio and “accept ” all points up 
to the desired confidence level. Then we scan through (/i, b) space to find 
the confidence region. As an example consider figure [I] which shows the 95% 
confidence region for the case x = 20, y = 7 and r = 1. 

Of course 6 is a nuisance parameter, and what we really want is a confidence 
interval for y. We can get that by projecting the region down onto the p-axis, 
but it is clear that such a method will suffer from over-coverage. Unfortunately 
no general method is known to extract a confidence interval from a confidence 
region in such a way that the confidence interval has correct coverage. 


2.2.2 Feldman-Cousins Profile Likelihood (FCPL) 

Because we are only interested in y we can try to eliminate the background 
rate b at some point during the calculations. One way to do this is to use 
the idea of profile likelihood already described above. Here when calculating 
the probabilities we replace the density /(x, y; y, b) by fpL{x, y] y) defined by 

fpL{x,y,y) = f{x,y,iJ,,b), where b is as in equation [d Now the method pro¬ 
ceeds exactly as the basic Feldman-Cousins method described above, except 
using fpL instead of /. 

One problem with this approach is that fpp is no longer a probability density, 
in fact Yi,x,y fpL{x,y; y) = oo. So we need to restrict our calculations to 0 < 
X < Mx and 0 < ?/ < My with and My large enough so that their choice 
does not effect the limits significantly. In the numerical studies shown below 
we always use M^ = My = 50, which we verified is large enough so that the 
effect on the limits is negligible. Moreover we need to normalize fpp so it is a 
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proper probability density with 


M, My 


x=0 y=0 


( 11 ) 


2.2.3 Cousins-Highland 

Another way to eliminate b is to nse a procednre hrst advocated by Consins 
and Highland . The idea is to eliminate a nnsiance parameter by integrating 
it ont. There are essentially two ways to do this: 

Feldman-Cousins-Cousins-Highland Option 1 (FCCHl) 

Here we replace the probability density /(x, y; p, h) with 


fcnix, y, p) := fix, y; p, b)db = 
^y-^^f^ipFbYbye-^^+-^^db = 

E;.o (3/1” /o“ = 

x\y\ ^n—0 \nj' 

where we made use of the fact that 


r{x+y-n+l) 
IP (^iJ^-rp+y-n+l- 



r(fc + i) 


( 12 ) 


(13) 


which in turn follows because the integrand is the density of a Gamma distri¬ 
bution with parameters k + 1 and a. T denotes the gamma function. 

Now limits are found the same way as before. Again we have the problem that 
J2x,y fcHix,y] p) = oo, and we proceed as in section 2.2.2. The normalized 
probabilities will be denoted by f^^j. 

Feldman-Cousins-Cousins-Highland Option 2 (FCCH2) 

In this version one hrst calculates limits T(a:; b) and 17(a:; b) for the signal rate 
p using the method of Feldman and Cousins for hxed background rate b, and 
then hnds limits 

Hx, y) = /o“ Lix] b)T^^e-^^db 
Uix,y) = fYUix-by^e-^^db 
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essentially weighting each limit by the corresponding Poisson probabilities. 
The extra factor r comes from the normalization = K 


2 . 2.4 Neyman Construction with Probability Ordering (NeyProb) 

There is yet another variation of this method: instead of using the likelihood 
ratio as the ordering principle we can simply use the probabilities One 
of the reasons Feldman and Cousins did not use this ordering was that it can 
lead to empty intervals. For example if we have r = 1, observe x = 0,?/ = 6 
and want to hnd 95% limits there is no /r > 0 that will have the point (0, 6) in 
the acceptance region. If this is deemed acceptable, or if it is known a priori 
that there will be more events in the signal region than are expected from 
background alone, this is a viable method. 


2.3 CLs 


A method that has been used extensively in HEP is the CLs method. Here in 
the case of a known background rate b one uses the test statistic 


P{X < x|/i + &) 
P{X < x\b) 


(15) 


which means one is testing specihcally Hq : = 0 vs Ha : /r > 0. Therefore 

this method always yields upper limits. 


CLs was hrst proposed in a special case by 
Read 
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Zech 18 and generalized by 


Even though it has very little grounding in Statistical theory it 
has become quite popular in HEP. The extension to our “On-Off ” model was 
done by Gan and Kass 1^, who used the Cousins-Highland prescription to 


show that an upper limit can be found by inverting a test based on the test 
statistic 


TcL,(K^,y,T) 




)/{n - k)\/{l + tY 

y' e;.o ( 


/{l + tY 


(16) 


and the (say) 95% upper limit is found by solving Tcls{.H) Hi' t) = 0.95. The 
derivation of equation 16 is very similar to the calculations done for fcH- 


2.3.1 Bayesian Methods 

The last class of methods we will consider are intervals derived using the 
Bayesian approach. Although we will hnd proper Bayesian credible intervals 










these will then be evaluated as standard frequentist conhdence intervals. He¬ 
lene used a Bayesian approach to calculate limits for the “On-Off ” prob¬ 
lem, although they modeled the background as a Gaussian rather than a Pois¬ 
son. 


As always in Bayesian Statistics we need to choose priors for fi and b. We 
will consider the following cases: flat priors, a fairly common choice in HEP, 
and Jeffrey’s non-informative prior, which in the case of a Poisson distribution 
with rate /i is given by vr(p) = 1/^/JI. We will consider all combinations of 
these priors. Lastly we will include another choice occasionally found in the 


literature, namely 7r(/i, 6) = 


constant. 


■yG+b 


This is Jeffreys prior when 6 is a known 


The last case does not allow for an analytic solution, and limits are found 
through numerical integration. For the hrst four we can handle all cases in 
one calculation by assuming priors of the form 7 r(A) = A“^, where p = 0 or 
1/2. With this we have the probability model 


X ~ Pois{fi + b), Y ~ Poisib) nip) = b ^ 7r(/i) = /i 
f{x,y,fi,b) = = (17) 

X, p = 0,1,..; /i, 6 > 0 

we begin by Ending the marginal distribution of x and y. 


m{x,y) = /o“ f(x,y;yb)7r(b)7r(y)df2db = 
^ /o“ (P^(b + db = 


ry 

x\y\ 

ry 

x\y\ 

i-y 

x\y\ 

i-y 

x[y\ 

yy 

x[y\ 


joc ^,-pg-(i+r)6 (joo y-^e-^dy) db = 

En=o ft) /o“ (/o“ db = 

E?:=o ft) r - n - 7 + l)db = 

E:=o ft)r(x - n - 7 + 1) /o“ = 

ft)r(x - n - 7+1) 


(18) 


Next we find the posterior density of y as the marginal of the joint posterior 
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density: 


y) = /o“ h\x, y)db = 

roc = 

myx^y) 

+ bri'-r^-'e-r-i^^-'Odb = 


(19) 


m(x,y)x\y] 


3 fd 


fx\,^x—n—'f r(i;+n—p+1) 

2-n=0 


(l+T-)y + 'n-p+l 


We also need the posterior distribution function F(/i|a;,|/): 


= /o^/(t|a;,|/)dt = 


y^x /x\ r(y+n-p+l) 

2^n=0 m{x^y)x\y\ \n/ 

y^x /x\ r(y+n-p+l) 

^n=0 m(x,^)x!^! \n) 

y^x /x\ r(y+n—p+1) 

^n=0 m(x,p)x!p! \n/ 


t^-^-^e-^dt = 
r(a; - n - 7 + 1) 

r(a; — n — 7 + l)Fr(/i; a; — n — 7 + 1,1) 


( 20 ) 


where Fr{.',a,/3) is the distribution function of a Gamma random variable 
with parameters {a,/3). 


Now we need to extract an interval from the posterior distribution. We will use 
the method of highest posterior density, which is the solution of the system of 
equations 

f{L\x,y) = f{U\x,y) 

F{U\x, y) - F{L\x, y) = I - a 


The advantage of this solution over the more common equal tail area solution 
is that this method yields a smooth transition from one sided to two sided 
intervals and therefore avoids the problem of flip-flopping, which is further 
discussed in the section on Performance. 

In the following we will denote the Bayesian methods by their priors, so for 
example 7r(/i, 6) = 1 is the method with flat priors on both y and 6, 7r(/i, b) = 
1/^/Jl uses Jeffrey’s prior on y and a flat prior on b etc. 


3 Performance of these Methods 


3.1 Coverage 
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The coverage Cov of a set of confidence intervals [L{x,y),U{x,y)\ is defined 
by 

Cov{y, b)=P {L{X, Y)<y< U{X, Y)\y, b) = 

(zzj 

12x=0 ^y=0 ^L{x,y),U{x,y)] (h)/(^) Vi h) 

where Ia{x) is the indicator function of the set A and f{x,y;y,b) is the 
probability density from equation 

A (1 — a) 100% conhdence interval is said to have coverage if for all (/i, 6) G 
[0, cx)) X [0, cx)) 

Cov{y, b) > 1 — a (23) 

For this paper we will restrict ourselves to the region of parameter space with 
(/i, 6) G [0,20] X [0,10], relevant for counting experiments with low statistics. 
In the case of larger values of y and b one would likely use some asymptotic 
methods as studied by Cowan et.ah [^. 

The experimenter might on occasion decide ahead of time that they will only 
calculate an upper limit, for example if theory suggests that the signal rate 
is very small or even 0. The distinction between upper limits and two-sided 
conhdence intervals is somewhat artihcial because an upper limit can always 
be viewed as a conhdence interval with L{x, y) = —oo for all x, y. Moreover the 
property of coverage applies equally to both. One important point, though, 
is that for a method that yields either one or the other the experimenter 
must decide before seeing the data which one he wants to use. Deciding this 
based on the observed data leads to the hip-hopping problem, which generally 
leads to under-coverage. All of the methods used in this paper have a smooth 
transition from upper limit to two-sided interval, except for CLs, which by 
design yields upper limits only. 

If at some point in parameter space we have Cov(y, b) > 1 — a the method 
is said to overcover. Overcoverage is “legal” in the sense that a method is 
still said to have coverage but is undesirable because it generally comes at the 
price of larger intervals. Unfortunately in the case of a discrete distribution 
such as the Poisson overcoverage at almost all points in parameter space is 
unavoidable. 

On the other hand if we have Cov{y, b) < 1 — a the method is said to un¬ 
dercover. This is a much bigger problem to the point of making the method 
useless. In practice, though, a small amount of undercoverage is generally 
deemed to be acceptable, and in fact as we shall see all the methods described 
here undercover at least a little in some part of parameter space. If one were 
to decide that any amount of undercoverage is unacceptable, one could pro¬ 
ceed as follows. Say the limits [L{x,y),U{x,y)] have been calculated to yield 
90% conhdence intervals, but at some point {y, b) the actual coverage is only 
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85%. Then calculating the limits at a higher nominal conhdence level will also 
increase the actual worst coverage, and there exists a nominal conhdence level 
so that the actual lowest coverage is the desired one. 


This is known in Statistics as the method of adjusted p-values. For an example 
see Aldor-Noiman et al. Q and for a general discussion see Rolke and Buja [^. 

Let’s begin with a graph of the coverage for the case b = 0.5, r = 1 and 90% 
conhdence intervals, shown in hgure We see surprisingly bad performances 
of FCCHl and FCPL. In the case of FCCHl the minimum occurs for /i = 
1.0 where the actual coverage is only 77%. This turns out to be due to the 
fact that for the case x = 0, y = 0 the upper limit of the 95% interval of 
FCCHl is only 1.45, so if we check coverage for (say) y + h = 1.5 this case is 
excluded, although /(O, 0; 1.0, 0.5) = 0.12, clearly the probability missing for 
good coverage. 


Similarly the (to) small limit of FCPL for x = O^y = 0 leads to bad un¬ 
dercoverage, this time at /i = 1.4. The coverage of the Bayesian methods 
with Jehrey’s prior on /i is also quite bad, mainly because these priors favor 
smaller values of p. Finally the prior 7 i{y,b) = 1/y/y + b leads to a method 
with coverage that is somewhat borderline. 


Figure IT shows the coverage for the case b = 5.0, r = 1 and 90% conhdence 
intervals. Here the worst method is FCCH2, with a true coverage of 42% if 
/i = 0.0! This is due to the fact that the averaging over the lower limits leads 
very quickly to a positive lower limit, for example ii x = 2,y = 0 ,t = 1 we 
get the 90% interval (0.09,4.29), and so for /r < 0.9 this case is rejected. 


Finally we will hnd the worst coverage of each method by searching over a 
grid on 0.5 < 6 < 10 and 0 < p < 20. The results are shown in table As 
we saw before FCCHl, FCCH2, FCPL, l/s/Jl, 1/y/JIb and l/y/y + b show 
some considerable under cover age. These methods will therefore be removed 
from further consideration. 


Of course one should repeat the above studies for other values of r and other 
conhdence levels. In the appendix we have the corresponding graphs and tables 
for the cases r = 0.5, 2 and 68% and 95% intervals. The results are very similar 
to those shown here. 


3.2 Other Considerations 


This leaves us with a choice of six methods. How do we decide among those? 
Here the experimenter is free to use any criterion he wishes, provided that the 
choice is made without consideration of the data. We already mentioned two. 
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namely avoiding the problem of flip-flopping and/or avoiding empty intervals. 
All (except CLs, which always yields upper limits) of the methods discussed 
here have a smooth transition from upper limits to two-sided intervals, and 
so flip-flopping is not an issue. The only method which could yield empty 
intervals is NeyProb, which might be eliminated from consideration for that 
reason. 

A very common criterion for the performance of conhdence intervals in Statis¬ 
tics is the expected mean length, dehned by 

OO OO 

EL{y, b) = y) - y)] ( 24 ) 

33=0 y=0 


Why short intervals are desirable is most easily seen in the case of upper limits, 
where it simply means a tighter bound on the parameter of interest. Even in 
the case of a two-sided interval, knowing that the parameter is most likely 
in the interval (say) (2.5,4.9) is better than just knowing it is in the interval 
(2.1, 5.3). In general, for confidence intervals the expected length plays a role 
similar to the concept of power in the case of hypothesis testing. 


It should be noted that the expected length is metric dependent. So a change 
in the parametrization of the problem might also change which method yields 
shortest expected length. 


In figure we show the expected length as a function of y for the case 
b = 0.5, r = 1 and 90% conhdence intervals, and in hgure 16 the same for the 
case b = 5.0. In both cases RLC has the shortest intervals for small fi and 
the Bayesian methods for larger ones. As one would expect the overcoverage 
observed for CLs, NeyProb and FC2D leads to larger expected intervals 


4 Conclusions 


We have studied the coverage and the expected length of the conhdence in¬ 
tervals generated by a number of methods for the “On-Oh ” problem. The 
intervals were derived using a variety of methodologies and include all those 
in common use today. We hnd that the RLC limits based on the prohle likeli¬ 
hood and the limits derived using the Bayesian methodology with a hat prior 
on the signal rate y are best, all having acceptable coverage and shortest ex¬ 
pected length. It is noteworthy that the oldest method in this study, namely 
the method implemented in MINUIT/MINOS, is still a strong contender even 
today, at least when used with some adjustments for the cases when x < y/r 
as is done in RLC. It should also be mentioned that just because a hat prior 
on fi leads to methods with good coverage for the “On-Oh ” problem studied 
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here, this does not necessarily mean that flat priors will always be the best 
choice. 

This study was possible because for this simple model we were able to find 
explicit formulas for the limits. It would obviously be desirable to do similar 
studies for more complicated models, for example including uncertainties in 
r, including efficiencies with their uncertainties as well extensions such as 
multiple channels. In those cases, though, deriving explicit formulas will be 
difficult if not impossible, and as soon as MC methods are needed to calculate 
limits the scope of any coverage study will be severely limited. Nevertheless 
we hope this study will provide some guidance as to which methods are most 
promising. 
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5 Appendix 


5.1 Confidence region for Feldman- Cousins 

Fig. 1. 2 dimensional 95% confidence region for if x = 20, y = 7 and r = 1.0. 
A 95% confidence interval for g is found by projecting region onto g axis 
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5.2 Craphs and Tables of Coverage and Expected Lengths 
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5.2.1 Case r = 0.5 and 68% Confidence Intervals 
Table 1 

Worst coverage of each method, for 0.5 < 6 < 10 and 0 < /r < 20. r = 0.5 and 68 
% confidence intervals 



RLC 

FCD2 

FCPL 

FCCHl 

FCCH2 

NeyProb 

CLs 

b 

3 

0.5 

0.5 

0.5 

10 

10 

0.5 

fj- 

8.6 

8.2 

1.2 

0.8 

20 

19.6 

6.5 

Coverage 

63.7 

86.2 

81.3 

72.2 

24.3 

81.6 

51.8 


1 


l/Vb 

l/y/fih 

l/\//x + 6 


b 

0.5 

10 

0.5 

0.5 

0.5 

d- 

4.1 

9 

4.5 

0 

3.3 

Coverage 

59 

61.9 

33.5 

39.3 

50.2 


Fig. 2. Coverage and Expected Lengths for for the case b = 0.5, r = 0.5 and 68% 
confidence intervals 
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Fig. 3. Coverage and Expected Lengths for for the case b = 3, t = 0.5 and 68% 
confidence intervals 
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Fig. 4. Coverage and Expected Lengths for for the case 6 = 5, r = 0.5 and 68% 
confidence intervals 
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5.2.2 Case r = 0.5 and 90% Confidence Intervals 
Table 2 

Worst coverage of each method, for 0.5 < 6 < 10 and 0 < /r < 20. r = 0.5 and 90 
% confidence intervals 



RLC 

FCD2 

FCPL 

FCCHl 

FCCH2 

NeyProb 

CLs 

b 

10 

10 

3.2 

0.5 

10 

10 

0.5 

fj- 

20 

0.4 

0.4 

1.6 

0 

20 

16.3 

Coverage 

87.8 

91.8 

87.8 

87.4 

21.1 

90.3 

83.8 


1 


l/Vb 


l/Vli + h 


b 

0.5 

10 

0.5 

0.5 

0.5 

d- 

8.2 

18.8 

9.4 

6.5 

7.8 

Coverage 

83.3 

86.1 

73.8 

78.7 

79.8 


Fig. 5. Coverage and Expected Lengths for for the case b = 0.5, r = 0.5 and 90% 
confidence intervals 
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Fig. 6. Coverage and Expected Lengths for for the case b = 3, t = 0.5 and 90% 
confidence intervals 
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Fig. 7. Coverage and Expected Lengths for for the case 6 = 5, r = 0.5 and 90% 
confidence intervals 
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5.2.3 Case r = 0.5 and 95% Confidence Intervals 
Table 3 

Worst coverage of each method, for 0.5 < 6 < 10 and 0 < /r < 20. r = 0.5 and 95 
% confidence intervals 



RLC 

FCD2 

FCPL 

FCCHl 

FCCH2 

NeyProb 

CLs 

b 

10 

9.8 

10 

10 

10 

10 

0.5 

fj- 

20 

0.4 

20 

20 

0 

20 

11.8 

Coverage 

92.6 

94.8 

92.6 

93.6 

25.6 

93.6 

90.9 


1 


l/Vb 


l/Vli + h 


b 

10 

10 

0.5 

10 

0.5 

d- 

20 

20 

6.9 

19.6 

11.4 

Coverage 

90.9 

91.6 

83.8 

88.2 

88.8 


Fig. 8. Coverage and Expected Lengths for for the case b = 0.5, r = 0.5 and 95% 
confidence intervals 
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Fig. 9. Coverage and Expected Lengths for for the case b = 3, t = 0.5 and 95% 
confidence intervals 
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Fig. 10. Coverage and Expected Lengths for for the case b = 5, t = 0.5 and 95% 
confidence intervals 



Coverage 

1 .0- 








0 5 10 15 20 

Expected Length 



0 5 10 15 20 



000 


.975 


.950 


.925 


.900 


0 


20 


— FCD2 

— FCPL 

— FCCH1 

— FCCH2 

— NeyProb 

— CLs 

— 1 

— 1/Vb 

— 1 /# 

— I/Vgb 

— 1 /i/p + b 


1 . 

0 . 

0 . 

0 . 

0 . 


25 



















5.2.4 Case r = 1 and 68% Confidence Intervals 
Table 4 

Worst coverage of each method, for 0.5 < 6 < 10 and 0 < /r < 20. r = 1 and 68 % 
confidence intervals 



RLC 

FCD2 

FCPL 

FCCHl 

FCCH2 

NeyProb 

CLs 

b 

1.3 

0.5 

0.5 

0.5 

10 

10 

0.5 

d 

1.6 

4.5 

0.8 

0.4 

0 

20 

7.3 

Coverage 

63.6 

79 

48.3 

43.1 

18 

72.8 

60.9 


1 


l/Vb 

\lyjfih 

l/\//i + 6 


b 

10 

8.6 

0.5 

0.5 

0.5 

d 

7.8 

6.9 

0 

2.9 

2.4 

Coverage 

60.9 

61.8 

39.3 

37.5 

51.1 


Fig. 11. Coverage and Expected Lengths for for the case b = 0.5, r = 1 and 68% 
confidence intervals 
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Fig. 12. Coverage and Expected Lengths for for the case 6 = 3, r = 1 and 68% 
confidence intervals 
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Fig. 13. Coverage and Expected Lengths for for the case 6 = 5, r = 1 and 68% 
confidence intervals 
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5.2.5 Case r = 1 and 90% Confidence Intervals 
Table 5 

Worst coverage of each method, for 0.5 < 6 < 10 and 0 < /r < 20. r = 1 and 90 % 
confidence intervals 



RLC 

FCD2 

FCPL 

FCCHl 

FCCH2 

NeyProb 

CLs 

b 

0.5 

10 

0.7 

0.5 

10 

10 

10 

d 

3.7 

0.4 

2.9 

1.2 

20 

20 

20 

Coverage 

87.7 

88 

83.4 

80 

28.8 

90.5 

86.8 


1 


l/Vb 

\lyjfih 

l/\//r + 6 


b 

2.1 

10 

0.5 

10 

0.5 

d 

9.8 

14.7 

6.9 

20 

6.9 

Coverage 

86.9 

87.4 

80.4 

81.3 

83.9 


Fig. 14. Coverage and Expected Lengths for for the case b = 0.5, r = 1 and 90% 
confidence intervals 
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Fig. 15. Coverage and Expected Lengths for for the case 6 = 3, r = 1 and 90% 
confidence intervals 
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Fig. 16. Coverage and Expected Lengths for for the case 6 = 5, r = 1 and 90% 
confidence intervals 
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5.2.6 Case r = 1 and 95% Confidence Intervals 
Table 6 

Worst coverage of each method, for 0.5 < 6 < 10 and 0 < /r < 20. r = 1 and 95 % 
confidence intervals 



RLC 

FCD2 

FCPL 

FCCHl 

FCCH2 

NeyProb 

CLs 

b 

10 

10 

0.5 

0.5 

10 

10 

10 

6 

20 

20 

2 

1.6 

19.6 

20 

20 

Coverage 

93.1 

95.9 

89.4 

86.8 

32.5 

93.9 

91.8 


1 


l/Vb 

\lyjfih 

l/\//i + 6 


b 

10 

10 

10 

10 

10 

d 

20 

20 

20 

20 

20 

Coverage 

91.3 

91.9 

87.9 

89 

90.7 


Fig. 17. Coverage and Expected Lengths for for the case b = 0.5, r = 1 and 95% 
confidence intervals 
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Fig. 18. Coverage and Expected Lengths for for the case 6 = 3, r = 1 and 95% 
confidence intervals 
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Fig. 19. Coverage and Expected Lengths for for the case 6 = 5, r = 1 and 95% 
confidence intervals 
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5.2.1 Case r = 2 and 68% Confidence Intervals 
Table 7 

Worst coverage of each method, for 0.5 < 6 < 10 and 0 < /r < 20. r = 2 and 68 % 
confidence intervals 



RLC 

FCD2 

FCPL 

FCCHl 

FCCH2 

NeyProb 

CLs 

b 

0.5 

10 

0.5 

0.5 

10 

10 

10 

d 

2.4 

0 

0.8 

0.4 

20 

20 

20 

Coverage 

62.8 

76.3 

52.8 

47 

23.9 

67.3 

65.5 


1 


l/Vb 

\lyjfih 

l/\//i + 6 


b 

7.9 

9.4 

0.5 

0.5 

0.5 

d 

6.1 

6.5 

0 

0 

2.9 

Coverage 

61.9 

62.2 

38.8 

38.7 

56.3 


Fig. 20. Coverage and Expected Lengths for for the case b = 0.5, r = 2 and 68% 
confidence intervals 
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Fig. 21. Coverage and Expected Lengths for for the case 6 = 3, r = 2 and 68% 
confidence intervals 
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Fig. 22. Coverage and Expected Lengths for for the case 6 = 5, r = 2 and 68% 
confidence intervals 
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5.2.8 Case r = 2 and 90% Confidence Intervals 
Table 8 

Worst coverage of each method, for 0.5 < 6 < 10 and 0 < /r < 20. r = 2 and 90 % 
confidence intervals 



RLC 

FCD2 

FCPL 

FCCHl 

FCCH2 

NeyProb 

CLs 

b 

0.5 

0.5 

0.5 

0.5 

10 

10 

10 

d 

2 

19.6 

1.6 

0.4 

20 

20 

20 

Coverage 

85.6 

93.6 

84.8 

56.8 

30.4 

88.1 

87.3 


1 


l/Vb 

\lyjfih 

l/\//i + 6 


b 

10 

7.7 

1.9 

9.4 

0.5 

d 

13.5 

12.2 

7.8 

11.8 

5.3 

Coverage 

87.4 

87.6 

79.7 

82.2 

84.6 


Fig. 23. Coverage and Expected Lengths for for the case b = 0.5, r = 2 and 90% 
confidence intervals 
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Fig. 24. Coverage and Expected Lengths for for the case 6 = 3, r = 2 and 90% 
confidence intervals 
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Fig. 25. Coverage and Expected Lengths for for the case 6 = 5, r = 2 and 90% 
confidence intervals 
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5.2.9 Case r = 2 and 95% Confidence Intervals 
Table 9 

Worst coverage of each method, for 0.5 < 6 < 10 and 0 < /r < 20. r = 2 and 95 % 
confidence intervals 



RLC 

FCD2 

FCPL 

FCCHl 

FCCH2 

NeyProb 

CLs 

b 

0.5 

10 

0.5 

0.5 

10 

10 

10 

9 

2.9 

20 

2.4 

0.8 

20 

20 

20 

Coverage 

93.3 

96.5 

91.4 

72.7 

33.8 

92.7 

92.3 


1 


l/Vb 

\lyjfih 

l/\//i + 6 


b 

10 

10 

10 

1.1 

10 

9 

20 

20 

20 

9.4 

20 

Coverage 

92.5 

92.7 

89.4 

89.6 

91.8 


Fig. 26. Coverage and Expected Lengths for for the case b = 0.5, r = 2 and 95% 
confidence intervals 
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Fig. 27. Coverage and Expected Lengths for for the case 6 = 3, r = 2 and 95% 
confidence intervals 
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Fig. 28. Coverage and Expected Lengths for for the case 6 = 5, r = 2 and 95% 
confidence intervals 
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